Lecture #10 


Big-O 
What's the big picture? 


Big-O is a technique that lets you quickly 
understand an algorithm's efficiency. 


“This function has a big-O of N? which means 
that if you pass in N items to it, it will take 
roughly N? steps to process its data." 


Big-O is not exact - instead, it gives rough 
estimates in terms of well-known functions 
like logo(N), N, N*loga(N), or Nê. 


We use the Big-O approach to quickly compare 
different algorithms and pick the best one. 


How Fast is That Algorithm? 


Sometimes we want to ask “How fast is that algorithm?" 


Or: “Which algorithm is 
faster, A or B?" 


Question: How can you measure the “speed” 
of an algorithm? 


How Fast is That a 


Right! We could measure the time it 
takes for an algorithm to run. 


But that has flaws! 
What are they? 


Carey: “My algorithm finished in 31 seconds." 
Cedric: “Mine finished in 30 seconds, it's better!" 
Carey: “Not so fast! How fast is your PC? Mine is 1GHZ." 


Cedric: “Err... Mine is 3GHZ." 
Carey: “Aha - so my algorithm is really almost 3x faster!" 


Cedric: “Sigh. Carey's right again." 


How Fast is That Algorithm? 


Ok, so simply measuring the run-time of an 
algorithm isn't really all that useful. 
What if instead we measure an algorithm based on: 


how many computer instructions it takes 
to solve a problem of a given size 


Carey: “My algorithm took 370 million instructions 
to sort 1,000 numbers.” 


Cedric: “Dude - you SUCK! Mine only took only 5 million 
instructions, it's better!" 


Carey: “Not so fast grasshopper! Mine might be slower on 
1,000 numbers, but what if we sort 1 million numbers?" 


Cedric: “Hmm. I don't know - I haven't tried." 


How Fast is That Algorithm? 


So just rating an algorithm based on how many steps it 
takes ona particular set of data doesn't tell us much. 


An algorithm might look efficient when applied to a small 
amount of data (e.g., 1,000 numbers) 


But really “blow chunks" when applied to a lot of data 
(e.g. 1 billion numbers) 


We'd like to understand how our algorithm 
performs under all circumstances! 


How Fast is That Algorithm? 


Hmmm. What else could we do? 


Right! What if we specify 
the number of instructions used by an algorithm 
as a function of the size of the input data. 


“I'm tryina to sort. N numbers." 
“Algorithm A takes 5N? instructions to do that." 
“Algorithm B takes_37,000-N instructions to do that." 


Now we can predict which algorithm will be faster 
for any value of N! 


How Fast is That Algorithm? 


“I'm trying to sort N numbers." 
“Algorithm A takes 5-N¢ instructions to do that." 
“Algorithm B takes 37,000-N instructions to do that." 


Ok, what if we're sorting 1,000 numbers: 
“Algorithm A takes 5M instructions." 
“Algorithm B takes 37M instructions." 

Ok, what if we're sorting 10,000 numbers: 

“Algorithm A takes 500M instructions." 


“Algorithm B takes 370M instructions." ay 


DN 


(1S 
Ok, what if we're sorting 1 million numbers: 


“Algorithm A takes 5 trillion instructions." _ 
“Algorithm B takes 37 billion instructions." 


How Fast is That Algorithm? 


“I'm trying to sort N numbers." 
“Algorithm A takes 5-N¢ instructions to do that." 
“Algorithm B takes 37,000-N instructions to do that." 


Cool! When we measure this way, we get two benefits: 
1. We can compare two algorithms for a given sized input. 


2. Wecan predict the performance of those algorithms 
when they are applied to less or more data. 


This is the idea behind the “Big-O" concept used in 
Computer Science. 


No, not Oprah. Let's learn the details! 


Big-O: The Concept 


The Big-O approach measures an algorithm 
by the gross number of steps that it requires 
to process an input of size N 
in the WORST CASE scenario. 


WORST ca 
ASE 
Survival Hanat ARIO 


We could be specific and say: 
“Algorithm X requires 5N?+3N+20 steps to process N items." 


But with Big-O, we 
ignore the coefficients and lower-order terms of the expression... 


So we'd say: 
“The Big-O of Algorithm X is N2." 


While less specific, this still gives us an overall 
impression of an algorithm's worst-case efficiency. 


Big-O: The Concept 


Big-O Idea: Use simple functions like log(n), n, n?, n log(n), 
n3, etc. to convey how many operations an algorithm must 
perform to process n items in the worst case. 


This is pronounced: . 


“oh of n squared" 


“That sorting algorithm is O(n), so to sort n=1000 items it 
requires roughly 1 million operations." 


“That sorting algorithm is O(n:logzn), so to sort n=1000 
items requires roughly 10,000 operations." 


This allows us to easily compare two different 
algorithms: 


“Algorithm A is O(n*), which is much slower than 
algorithm B which is O(n:logən)." 
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Big-O 
So how do we compute the Big-O of a function? 


First, we need to determine the number of operations 
an algorithm performs. Let's call this f(n). 


By operations, we mean any of the following: 


1. Accessing an item (e.g. an item in an array) 
2. Evaluating a mathematical expression 
3. Traversing a single link in a linked list, etc... 


Let's see how to evaluate the number of operations for 
a simple example... 


Big-O - How to Compute f(n) 


Compute f(n), the # of 


int arr[n]in]; critical operations, that this 
for (inti=0;i<n; i++) algorithm performs? 
for (int j=0; j< n; j++) 
arr[i][j] = O; f(n)=1+n+n+n+n? +n? +n? 


f(n) = 3n? +3n +1 
. Our algorithm initializes the value of i once. 
. Our algorithm performs n comparisons between i and n. 
. Our algorithm increments the variable /, n times. 
Our algorithm initializes the value of j, n different times. 
. Our algorithm performs n? comparisons between j and n. 
. Our algorithm increments the variable j, n? times. 
. Our algorithm sets arr[i][j]'s value n? times. 


Now that we have f(n), we can compute our algorithm's Big-O. 


Big-O - The Complete Approach 


Here are the steps to compute the Big-O of an algorithm: 


1. Determine how many steps f(n) an algorithm requires 
to solve a problem, in terms of the number of items n. 


2. Keep the most significant term of that function and 
throw away the rest. For example: 
a. f(n) = 3n@+3n+1 becomes f(n) = 3n? 
b. f(n) = 2n log(n) + 3n becomes f(n) = 2n log(n) 


3. Now remove any constant multiplier from the function: 
a. f(n) = 3n? becomes f(n) = n? 
b. f(n) = 2n log(n) becomes f(n) = n log(n) 


4. This gives you your “big oh": 
a. f(n) = 3n2+3n+1 is therefore O(n?) 
b. f(n) = 2n log(n) + 3n is therefore O(n log(n)) 


Big-O Simplification 


Actually, if you think about it, there's no need to compute 
the exact f(n) of an algorithm... 


Why? Because we end up throwing away all of the lower- 
order terms and coefficients anyway! 


All you need to do is focus on the most frequently 
occurring operation(s) to save time! 


int arr[n][n]; 


Oe ad f(n) = n? 
O; i <n; j++) Our algorithm is O(n°). 


TIE O; 


Big-O 
So if I say: “This algorithm is O(n?).” 


I really mean: “To process n items, this algorithm requires 
roughly n? operations." 


By using only the most 
significant term 
(e.g. n? from 2n?+3nt1) 


wo oO D> 
(æ) a (a) 
Q Q Q 
(æ) (æ) (æ) 


We can quickly obtain a 
rough approximation 


Number of operations 
=< = N N 
fo) (Sj (%) o 
fo) (2) QO (o>) 
O O O O 


of how many steps our 
algorithm will take 


a 
(e) 
(æ) 


e ee hen 


1 10 20 30 40 50 60 70 80 90 100 110 


Number of items, n 


oO 


to process n items. 


Big-O Complexity 


logan N nlog,n n? n? Jn 
3 10 30 100 1000 1000 
6 100 600 10,000 1,000,000 10% 
9 1,000 9,000 1,000,000 1,000,000,000 | 103° 
13 10,000 130,000 100,000,000 | 1074 wow! 
16 100,000 1,600,000 1019 1015 wow! 
19 1,000,000 |19,000,000 | 10/2 1018 wow! 


What if you wanted to use an O(n?) algorithm to sort a 


million numbers? Your algorithm would require roughly 


1,000 ,000,000 000,000,000 steps! 
But an O(n log2(n)) algorithm would use only 19,000,000 steps! 


Big-O Complexity 
1,000,000 ,000,000,000,000 vs 19,000,000! 


GREAT PROGRAMMERS know that the choice of 
algorithm makes all the difference in the world. 


NOT-SO-GREAT programmers think that you can tweak a 
poor algorithm to make it better! 


Say you improve an O(n?) algorithm from f(n) = 5n? steps to f(n) = 1.5n° 
steps. For n=1,000,000, that reduces the number of steps from 
5 000,000,000 000,000,000 to 1,500,000,000,000,000,000. 


(Big deal... so it'll take 1.5 years to run instead of 5 years) 


However, if you can find an algorithm that's O(n-log n) steps, say 
f(n) = 10-n log n, you can solve the problem in 190,000,000 steps. 


(Which will take just a few seconds or less on a modern PC!) 


20 


Don't be a Big-O NUT! 


When you're writing a program that operates on a large 
number of items, evaluating Big-O is key. It can mean the 
difference between a usable program and an unusable one. 


But what if you have a small number of items, e.g. n<50? 


4000 + 


In this case, all of your 
algorithms require only a 
small number of steps. 


3500 


N N wo 
oO oO j=) 
jo) Q Q 
O j=) Oo 


In such situations (when 


Number of operations 
— 
o1 
Q 
O 


you know n is small), 


forget which Big-O is lave 
better and choose the 500 
easiest-to-program 1 j 10 20 30 40 50 60 70 80 90 100 110 
algorithm. It'll save you Number of items, n 


lots of headaches. 
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for (int i = 0; i< n; i+=2 ) 
sum++, 


for (int i = 0; i< q; i++) 
for (int j=0;j<q; j++) 
SuM++; 


for (int i = O; i< n; i++) 
for (int j = 0; j< n*n; j++) 
SUM++,; 


k=n: 
while (k > 1) 
{ 


sum++; 
k = k/2; 
} 


Find the Big-O Challenge, Part 1 


for (int izO ; i< n ; i++) 


int k =n; 
while (k > 1) 
SUM++, 
k = k/3; 
void foo( ) 
int i, sum = O; 
er bari 
for (i=0 ; i < n*n ; i++) o 
sum += i; (N6501 xN)O 
ne 
s s ‘ eNJO 
for (i=O ; i < n*n*n ; i++) (oo 
A NJO 
SUM += I; $4614 
Of 442| ‘Wolsog 
} o4 do, woud 
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Find the Big-O Challenge, Part 1a 


int searchArray(int arr[], int n, int forValue) 
for (int i = 0; i< n; i++) 


if (arr[i] == forValue) 


return i; 
} 
return -1; // not found 
} 
void addItemToEndOfArray(int arr[], int &n, int addMe) 

‘SIN 
arr[n] = addMe; eoor 
n=n+1; uo dos ouo 

} +snf - (2)0 40 (1)O 

(N)O 


:0440q 
o4 do, woud 


Big-O... my 


Sometimes you'll run into an algorithm 
that isn't so clear-cut. For example, 
what's the Big-O of mystery? 


It's clear that the outer loop runs n 
times, but what about the inner loop? 


When q = O, the inner loop runs O iterations. So what's the Big-O? 
When q = 1, the inner loop runs 1 iteration. 2 
When q = 2, the inner loop runs 2 iterations. f(n) = av-n 

2 2 


O(n?) 


When q = n-1, the inner loop runs n-1 iterations. 


So the cout statement will run a total of: 
O times + 1 time + 2 times + 3 times + ... + n-1 times 


And if you recall a clever trick, this is equal to: ™ i. 
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Big-O: Such Ugly Math! © 
But we're not Math geeks... We're CS geeks! So here's a 
way to address these situations without formulas! 


Step 1: 

Locate all loops that don't run for a fixed number of iterations and 
determine the maximum number of iterations each loop could run for. 
Step 2: 

Turn these loops into loops with a fixed number of iterations, using 
their maximum possible iteration count. 
Step 3: 
Finally, do your Big-O analysis. 


O(n?) n O(n?) n*n 


for (int j = 0; j< n; j++) 
for (int k = 0; k< j; k++) 
SUM++, 


for (int i = O; i < q*q; i++ ) 
for (int j= 0; j< i: j++) 
SUM++; 


for (int i = O; i< n; i++) 
for (int j = 0; j < i*i; j++) 
for (int k = 0; k< j; k++) 
sum++, 


for (int i = 0; i< p; i++) 
for (int j = 0; j< i*i; j++) 
for (int k = 0; k< i; k++) 
sum++, 


Find the Big-O Challenge, Part 2 


for (int i= 0; i< n; i++) 
{ 


Circ arr[n]; 
arr[i].setRadius(i); 


for (int i20; i< n; i++) 
{ 
int k= i; 
while (k> 1) 
{ 


(N7601 xN)O 
jua4! dad uo ‘SaM! 


sum+t+; N 4042Nu4SU02 244 
S||DD SW24! N Jo AduuD 


k = k/2: uv buijonu,suoo {Dy} 


} 42640} 4 uop - (2zN)O 


} (cN)O 


‘446i 04 442] 
'wo44oq of do} woud 
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Big-O for Multi-input Algorithms 


The number of people, p, and Often, an algorithm will The number of CS 
the number of foods, f, are operate on two (or more) students, c, 
completely independent. independent data sets, each and EE students, e, are 
of a different size. also independent. 
void buffet(string people[], tnt p, void tinder(string csmajors[], int c, 
string foods[], int f) string eemajors[], int e) 
{ { 
int i, j; for (int i=0; i< c ;i++) 
for (int j=0; j < c :j++) 
for (i=0; i< p ;i++) cout << csmajors[i] << “ dates “` 
for (j=0; j < f :j++) << csmajors[j] << endl; 
cout << people[i] << “ ate `“ ee ae 
<< foods[j] << endl: K for (int k=O; k< e skt) l : 
cout << eemajors[k]<<" sits at home:; 
} * t majors[k]<<" sits at homes; 
OX 


In these cases, when we What? O(c? + e) and not just O(c?) ? 


compute the algorithm's C +| n As 
Big-O, we must to take into E a 


account both independent 
sizes. 


bigger than e, we must include both 
independent variables in our Big-O! 
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| Big-O for Multi-input Algorithms 


Why must we include both variables in the Big-O 
(even if one is higher-order than the other)? 


Because either variable could dominate the other! 


ALERT! But... don't forget - 
you still must eliminate lower- 
order terms for each 
independent variable! 


void tinder(string csmajors[], int c, 
string eemajors[], int e) 
{ 


for (int i=0; i < c ;i++) 
for (int j=0; j < c ;j++ 
cout << csmajors[i] << “ dates “ 


<< csmajors[j] << andl; 


void tinder(string csmajors[], int c, 
string eemajors[], int e) 


for (int k=O; k< e ;k++) { ae 
cout << eemajors[k]<< " sitat home" for (int i=0; i < c ;i++) c2 iterations. 
} O for (int j=0; j< c j++) 
Vv x cout << csmajors[i] << “ dates “ 


(© << csmajors[j] << endl; 
O for (int m=0; m < c ; m++) 
> cout << csmajors[m] << “ is stilt a nerd”; 


© iterations. 
ie x for (int k=O; k < e ;k++) e it 
cout << eemajors[k]<< " sits at home" 
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Find the Big-O Challenge, Part 3 


void bar( int n, int q ) 
{ 


for (int i=O;i< n*n; i++) 


for (int j = O; j < q; j++) 
cout << “I love CS!"; 


void bletch( int n, int q ) 


for (int izO ; i< n ; i++) 


|" 


cout << “Muahahaha!"; 


for (int izO ; i < q*q; i++) 
cout << “Vomit!": 


void burp( int n) 
{ 


for (int i20; i< n; i++) 
cout << “Muahahaha!"; 


for (int i=O;i< n*n; i++) 
cout << “Vomit!": 


void barf( int n, int q ) 
for (int i=O ; i< n ; i++) 
if (i == n/2) 
for (int k = 0; k< q; k++) 
cout << “Muahahaha!": 


} 
else 


cout << “Burp!"; 


jSW4! N 244 40 |}D uo 
dooj uauu! əy4 unu ou 
op 2M 'Z/u o4 jonbə sı 
ı uaym ‘aouo 4snf doo] 
Jauu! au} unu Ajuo aM 
4DYL 240U - (© + N)O 

(2N)O 


(20 +N)O 

(o.2N)O 

:44614 of 442] 
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The STL and Big-O 


Remember the STL - stacks, queues, sets, vectors, lists and maps? 
Well, these classes use algorithms to get things done... 
And these algorithms have Big-Os too! 


void inDict(set<string> & d, string w) For example, if we want to 
search for a word ina set that 
if (d.find(w) == dend()) — i contains n words (a dictionary), 
cout << w << ` isn't in dictionary!": it requires O( logo(n) ) steps! 
void otherFunc(vector<int> & vec) But if we want to add a value 
to the end of a vector holding 
vec.push_back(42), n items, it takes just one step, 
vec.erase( vec.begin() ); so it's O( 1 )! 


} 


And if we want to delete the 1st value from a vector containing 
n items, it takes a whopping n steps, making it O(n)! 


“4 N 


Well, to search a set of 
n items for a single value 
requires log2(n) steps... 


~ 


It's important to understand the Big=O 
of each operation 
(e.g. push_back, erase) for each STL 
class (e.g., list, vector)... 


.. because without knowing the Big-Os of 
the STL classes, 


we can't compute the Big-O of code that 
uses the STL classes! 


For example... 


If we write a loop of our own that runs D 
times... 


And each iteration of our loop searches for 
an item in a set holding n items... 
Then what's the Big-O of our loop??? 


Then the Big-O of our whole 
loop would be O( D * log(n) ). 


The STL and Big-O 


And we repeat this 
search operation D 
different times... 


cout << w << ` isn 


for (int i=O; i < D ; i++) 
inDict( dict, doc[i] ); 


= Computing the Big-O of Algorithms that use STL 
What is the Big-O of the loop in terms of q? 


our loop runs 


q times... |. And each time our loop > 
runs, we: 
printNums( vector<int> 1. Access an item: the 
cost of accessing an item 
in a vector is O(1) - so 
we'll remember that! _/ 


Ok, our vector 


And as such, 
contains q values... 


int q = v.size(); 


for (int i= O;i <q; i++ 
{ 


i = s ; 2. Erase the first item in the 

ts < a ane Mk vector: the Big-O of erasing the 

v.erase( J begin() j at first item in a vector with q items 
sa .erase( v.beg ene is O(a). 


Á 


} 


} 


v.push_back(a); +L add it to 4 3. Add the item to the end of the 


end vector: the Big-O of adding an item 
to the end of a vector is O(1). 


The STL vector 


Total steps performed during our loop: eet at he top ele 


q* (1 +qg + 1) Insert an the end: 
Delete an item from top/middle: 
So our total Big-O is: Delete an item from the end: 
O(q?) Access an item: 


Finding an item: 


O(n) 
O(1) 
O(n) 
O(1) 
O(1) 
O(n) 


STL and Big Oh Cheat Sheet 


When describing the Big-O of each operation (e.g. insert) on a container (e.g., a vector) 
below, we assume that the container holds n items when the operation is performed. 


Name: list 
Purpose: Linked list 
Usage:  list<int> x; x.push_back(5); 


Inserting an item (top, middle*, or bottom): 


Deleting an item (top, middle*, or bottom): 
Accessing an item (top or bottom): 
Accessing an item (middle): 

Finding an item: 

*But to get to the middle, you may have to 


first iterate through X items, at cost O(x) 


O(1) 
O(1) 
O(1) 
O(n) 
O(n) 


Name: vector 
Purpose: A resizable array 


Usage: vector<int> v; v.push_back(42); 
Inserting an item (top, or middle): O(n) 
Inserting an item (bottom): O(1) 
Deleting an item (top, or middle): O(n) 
Deleting an item (bottom): O(1) 


Accessing an item (top, middle, or bottom): O(1) 
Finding an item: O(n) 


Name: set 

Purpose: Maintains a set of unique items 
Usage: set<string> s; s.insert("Ack!"); 
Inserting a new item: O(logon) 

Finding an item: O(logzn) 

Deleting an item: O(logzn) 


Name: queue and stack 
Purpose: Classic stack/queue 
Usage: queue<long> q; q.push(5); 
Inserting a new item: O(1) 
Popping an item: O(1) 
Examining the top: O(1) 


Name: map 

Purpose: Maps one item to another 
Usage: map<int,string> m; m[10] = “Bill”: 
Inserting a new item: O(logon) 

Finding an item: O(logzn) 

Deleting an item: O(logzn) 


If instead of holding n items, a container holds 


p items, then just replace “n” with “p 


Wall 


when you 
do your analysis. 


~ Computing the Big-O of Algorithms that use STL 


When evaluating STL-based algorithms, first determine the 


maximum # of items each container could possibly hold.. 


Then do your Big-O analysis under the assumption that 
each container always holds exactly this number of items. 


Ok, I'll assume that our set always has q 
items in it. That means that each time 
we insert an item into our set it takes 


A Ok. Well, after the loop » 
finishes, our set will hold 
q values... 


This is the maximum # of 
ee values it can hold. d 


log2(q) steps. 
C ai i p E if our loop runs a total ro 
q iterations... and each 
: iteration we insert at a cost 
- main( ) of log2(q) into our set... Then 
, our total cost is q * logż2(q) 
set< int > nums; ey 
for (int i=0; i < q ; i ++) 
nums.insert( i ); 
} 
And that's the correct 
| The STL set 
answer: Inserting a new item: O(logan) 
Finding an item: O(logan) 
Deleting an item: O(logan) 
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Find the Big-O Challenge, Part 4 


See if you can figure out the Big-O of these 


functions which use the STL! 


(Well. assuming sah 
vector has p items 
and we delete one 
item per iteration, 
our loop runs for p 


// Assume p items in vector v 
void clearFromFront(vector<int> &v) 


while ( v.size() > 0) 
{ 


oa 


Delete an item from top/middle: O(n) 
Delete an item from the end: O 

Access an item: 
Finding an item: 


So there are p total N 
loop iterations, and 
during each iteration, 
we perform p steps to 
delete the first item: 


Dele, Cost of deleting first item: O(p) n) 


steps. v.erase( v.begin() ); // erase 14t ite O(p? 
Ooo teso ( v.begin() ) D 
} 
Hints: “ During each step, we delete the 
vector's first item. 
The STL vector Let's assume the vector always 
Insert at the top/middle: O(n) Inse has p items, no matter what. ) 
Insert an the end: O(1) | Find 


O(1) 
O(n) 


a N 


So we perform q° total loop 
iterations, and during each 
iteration, we perform log2(q°) 
steps to insert an item: 


C O(q*"loga(q*)) P 


a i 

// Assume s starts out empty Ok, our loop runs 

void addItems(set<int> &s, int q) | through q? total 
iterations. 

for (int i=0; i < q*q; i++) \ d 

s.insert(i); 

} And the cost of adding a 

As before, to compute the cost of eo E ae ue q 
an STL operation, we assume the A eC eee 

set always holds its max # of items. 


In this case, the set will eventually 


hold q? items - that's our max! Inding an item: g 
Deleting an item: O(logan) 
Delete an item from the end: O(1) 
Access an item: O(1) 


Finding an item: O(n) 


37 


a A 


So there are z total loop 
iterations, and during each 
iteration, we perform 1 step 
to delete the last item: 
O(z) 


d 


T assuming our 
vector has z items 
and we delete one 
item per iteration, 
our loop runs for z 
steps. 
~ 
void clearFromBack(vector<int> &v) 
while ( v.size() > 0) 
} 
} 
Hint 
The STL vector 
Insert at the top/middle: O(n) 
Insert an the end: O(1) 
/mi 


O(1) 
@ 


Finding an item: O(n) 


_/ // Assume z items in vector -ai 


v.pop_back(); // erase last item 


During each iteration, we delete >, 
the vector's last item. 


Let's assume the vector always has z 


items, no matter what. 


Cost of deleting last item: O(1) / 
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Find the Big-O Challenge, Part 5 


— O 
{1,9,..} I have a vector of sets of ints: 
(eee) vectors set<int> > v; 
4; 8 . š 
; A You may assume vector v has N total sets in it. 
4 You may assume that each set has an avg of Q items. 
{-10,1,..} Questions: 


#1 What is the Big-O of determining if the first set, v[O], 
contains the value 7? 


#2 What is the Big-O of determining if any set in v has a value of 7? 
#3 What is the Big-O of determining the number of even values in all of v? 


#4 What is the Big-O of finding the first set with a value of 7 and then counting 
the number of even values in that set? 


© + (©)260].N vH 


xN € 
(©)260|,N Z# 
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Space Complexity 


Space complexity is the big-o of how much storage your 
algorithm uses, as a function of the data input size, n. 


void reverse(int array[], int n) void reverse(int array[], int n) 


Uses just , no 


int tmp, |. matter how big the array is! int *tmparr = new int[n]; 


i= Ori De Pe ‘ee or (int i = O; i< n; ++i) 

4 (i Sie 7) Py ) tmparr[n-i-1] = array[i]; 
tmp = array[i]; ` =e S for (int i = 0; i < n; ++i) 
array[i] = array[n-i-1]; array[i] = tmparr[i]; 


-j> 
e eae Sai delete [] tmparr; 


Space Complexity: Space Complexity: 
or 


This function uses just 
two 4-byte memory 
slots no matter how 


Space Complexity 


big n is. Be careful - space 
TO ; // prints from n down-to O 
S Complexity: 
ies Ba al! complexity ook 20 // with recursion! 
tricky with recursion! W : E 
i se 7pvoid printNums(int n 
I 
/ wt 
// prints from n down-to O / / { if (n< 0) return; 
// without recursion! /V in at 
cout << n << “\n"; 
void printNums(int n) J { printNums(n-1); 
} 
: m { =P Pri INUMOUI I), 
int 1; } 
—>for (izn; i >= O; i--) —p printiINUmMsS(N-L), 
cout << i << “\n"'; } 


-F pI TIITTNUIMNIOUIV Ly, 


The recursive version 
creates a whole new 
variable for each of the 
n levels of recursion! 


int main() 


{ 
—> printNums(1000); 
} 


t main() 


Space Complexity: O(n) printNums(1000); 


41 


Inefficient Sorting Algos. 


£ X 
complexities of \ 
different 
sorting 
algorithms is 


Inefficient Sorting Algorithms 
What's the big picture? 


Sorting is the process of ordering a bunch of Bee 
values, e.g., from smallest to largest. 4 


There are a handful of “inefficient” 
algorithms to do sorting, like selection sort, 
insertion sort and bubble sort. 


These algorithms generally require O(N*) 
steps to order N values. They're suuuper slow. N 


Why slow? They compare every item to every 
other item, swapping out-of-order items. 


Sorting! 


Sorting is the process of ordering a bunch of items based 
on one or more rules, subject to one or more constraints... 


Items - what are we sorting, 

and how many are there? 

- Strings, numbers, student records, 
C++ objects (e.g., Circles, Robots) 

- Thousands, millions or trillions? 


Rules - how do we order them? 
- Ascending J vs. Descending N order 
- Based on Circle radius? Student GPA? 
- Based on multiple criteria, e.g.: 

by last name, then first name 


Constraints? 
- Are the items in RAM or on disk? 
- Is the data in an array or a linked list? 


Carey's 2 Rules 
of Sorting 


Rule #1: 


Dont choose a sorting algorithm until you 
understand the requirements of your problem. 


Rule #2: 


Always choose the simplest sorting algorithm 
possible that meets your requirements. 


45 


The Sélection Sort 


- Look at all N books, select the 
shortest book 


- Swap this with the first book 

- Look at the remaining N-1 
books, and select the shortest 

- Swap this book with the 
second book 

- Look at the remaining N-2 
books, and select the shortest 


- Swap this book with the 
third book and so on... 


So, is our sort efficient? 


If we have N books, how many 
steps does it take to sort them? 


Let's assume a step is any time we 
either swap a book or 
point our finger at a book. 
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The Selection Sort- Speed 


- Look at all N books, select the 
shortest book 


1 step > Swap this with the first book 


- Look at the remaining N-1 
books, and select the shortest 


1 step > Swap this book with the 
second book 


- Look at the remaining N-2 
books, and select the shortest 


1 step > Swap this book with the Now if you remember your 

third book and so on... ee 
(and so on) N+N-1+N-2+..+2+1 

(and so on) is equal to... 
So this comes to: 
* 

N swap steps = me 
PLUS So Selection Sort is O(N2), or, for N 
N+N-1+N-24+..+241 books, you need roughly N? steps to 


steps to find the smallest item sort them. 
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Selection Sort - Better or Worse? 


Are there any kinds of input data where Selection 
Sort is either more or less efficient? 


For example, what if all of the 
books are mostly in order 
before our sort starts? 


void selectSort(shelf of N books) 
{ 
for i=1toN 
{ 
find the smallest book 
between slots i and N 


swap this smallest book 
with book i; 


No! Selection sort 
takes just as many 
steps either way! 
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The Se 


And here's the C++ 
Source code to sort a 
bunch of numbers... 


a” T 


Selection Sort Questions 


Can Selection Sort be applied easily to 
sort items within a linked list? 


Is Selection Sort “stable” or “unstable”? 


d 


void selectionSort (int A[], 


{ 


int n) 


for (int i = 0; i < n; i++) ]_ For each of the n array 


{ 
int minIndex 
for (int j 
{ 


=n. 


if (A[j] < A[minIndex]) 


minIndex Jy 
} 
swap (A[i], A[minIndex]) 


itl; J]J < ü; JTT) 


elements... 


Locate the smallest item 
— in the array between the 
ith slot and slot n-1. 


_ Swap the smallest item 
found with slot A[i]. 


e 
1 
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What's a Stable Sort? 


Imagine that N old people line up to buy laxatives at a drugstore. 


And the drugstore wants to sort them and serve them based on urgency. 


The drugstore needs to pick a sort algorithm to re-order the guests. 
They can choose between a “stable” sort or an “unstable” sort. 


An “unstable” sorting algorithm re-orders the items without taking into account their 
initial ordering. 
A “stable” sorting algorithm does take into account the initial 
ordering when sorting, maintaining the order of similar-valued items. 


As you solve problems (in class or at work) you should choose 
your sort depending on whether stability is important. 
If you forget the concept, just remember the laxatives! © 


People in line Unstable Sort Results Stable Sort Results 
Ebeneezer - 8 days Steve - 8 days Ebeneezer - 8 days 
Carey - 5 days Vicki - 8 days Steve - 8 days 
David - 2 days Ebeneezer - 8 days Vicki - 8 days 
Michael - 4 days Andrea - 5 days Carey - 5 days 
Steve - 8 days Carey - 5 days Andrea - 5 days 
Vicki - 8 days Michael - 4 days Michael - 4 days 


Andrea - 5 days David - 2 days David - 2 days 
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The Se 


And here's the C++ source 
code to sort a bunch of 
numbers... 


a” © 


Selection Sort Questions 


Can Selection Sort be applied easily to 
sort items within a linked list? 


Is Selection Sort “stable” or “unstable”? 


When might you use Selection Sort? d 


void selectionSort (int A[], 
{ 
for (int i 
{ 
int minIndex 
for (int j 
{ 


1? 


minin = j; 
} 
swap (A[i], A[minIndex] ) 


O; i < n; itt) 


itl; J < ü; JT) 


if (A[j] < A[minIndex]) 


int n) r. >» 
Here's a hint - consider 


thisarPay: 
10/10] 1 
When Selection Sort 


finds the 1, it swaps it 
with the first 10. 


Then our array ends up 


like this: 


7 


1|10|10 


The Insertion Sort 


Well, we couldn't just teach 
you one sort, right? 


Let's learn another'! 


The insertion sort is probably the 
most common way... 


to sort playing cards! 


(But I'll still explain the sort with 
library books) 
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The ANGETAN Sort 


Let's focus on the first two 
books - ignore the rest. 
- If the last book in this set is 
in the wrong order 
e Remove it from the shelf 
e Shift the book before it 
to the right 
e Insert our book into the 
proper slot 


Great! Now our first two 
books are in sorted order 
(ignoring the others) 


The Insertion Sort 


Ok, now focus on the first 
three books - ignore the rest. 
- If the last book in this set is 
in the wrong order 
- Remove it from the shelf 
° Shift the books before it 
to the right, as necessary 
- Insert our book into the 
proper slot 


Great! Now our first three 
books are in sorted order 
(ignoring the others) 


The lapel Sort 


Ok, now focus on the first 
four books - ignore the rest. 
- If the last book in this set is 
in the wrong order 
- Remove it from the shelf 
- Shift the books before it 
to the right, as necessary 
- Insert our book into the 
proper slot 


Great! Now our first four 
books are in sorted order! 


We just keep repeating this 
process until the entire shelf 
is sorted! 


i The Insertion Sort 


So what's the complete algorithm? E 
Start with set size s = 2 


While there are still books to sort: 
- Focus on the first s books 


- If the last book in this set is 
in the wrong order 


- Remove it from the shelf 


- Shift the books before it 
to the right, as necessary 


- Insert our book into the 
proper slot 


‘325+ ] 
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The Insertion Sort - Speed 


So what's the Big-O of our Insertion Sort? 


During each round of the algorithm 
we consider a larger set of books. 


During the first round, we may need to 
shift up to one book to find the right spot. 


1 step in round 1 


During the second round, we may need to + 2 steps in round 2 


shift up to two books to find the right spot. 


e.. + N-1 steps in last rnd 
During the last round, we may need to = roughly N? steps 


shift up to N-1 books to find the right spot. 


Thus, Insertion Sort 
is O(N*), and is 
generally quite slow! 
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Insertion Sort - Better or Worse? 


Are there any kinds of 
input data where 
Insertion Sort is either 
more or less efficient? 


Any ideas? 


Right! If all books are 
in the proper order... 


then Insertion Sort Conversely, a perfectly 


never needs to do any shifting! mis-ordered set of books 
is the worst case. 


In this case, it just takes roughly Bince eververouna 
~N steps to sort the array! O(N) requires the 
maximum shifts! 


" The Insertio 


And here's the C++ version 
which sorts an array in 
ascending order! < 


void insertionSort(int A[], int n) 


for(int s = 2; s <= n; s++) 


Insertion Sort Questions 


Can Insertion Sort be applied easily to 


sort items within a linked list? 
Is Insertion Sort a “stable” sort? 
When might you use Insertion Sort? 


N 


d 


Focus on successively larger 
prefixes of the array. Start 
with the first s=2 elements, 
then the first s=3 elements... 


{ 
imt SOE = Ase +} Make a copy of the last val in the 


inti=s- 2; 
while (i >= O && sortMe < A[i]) 


Ali+1] = ALi] 


} 


current set - this opens up a slot 
in the array for us to shift items! 


Shift the values in the focus 
region right until we find the 
proper slot for sortMe. 


Store the sortMe value into the 
Ali+1] = sortMe, See vacated slot. 
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Everyone loves tow » 
make fun of the: 


a CP OD “i 
But it's actually 
ba S So rt quite simple... And 
i O sometimes simple 
is good! 


Ok, what's the algorithm? 


Start at the top element of your array 


Compare the first two elements: A[O] and A[1] 
If they're out of order, then swap them 


Then advance one element in your array 
Compare these two elements: A[1] and A[2] 
If they're out of order, swap them 


Repeat this process of comparing A[j] with A[j+1] and swapping if they're 
out of order until you hit the end of the array 


When you hit the end of the array, if you made at least one swap on your 
way down, then start back at the top and repeat the whole process again! 
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Ok, so Bubble Sort Bubble Sort Speed 


has a bad wrap. | 
During each pass, we compare every element 


Question: with its successor (and possibly swap each). 
How fast is it? | 
or That requires about N steps. 
If we did even one swap, we need to repeat 
Just like the whole process again from the top. 
Insertion . 
Sort, What's the worst case? How many times 
Bubble Sort might we have to repeat the process? 
is really 
efficient on Right! We might have to repeat this 
pre-sorted entire process N times. 
arrays and ; ; 
linked lists! N passes of N “bubbles” = N? 


Ok, so Bubble Sort is O(N?)... 
But can it ever run faster in certain cases? 


The Bubble Sort 


Bubble Sort Questions > 


Can Bubble Sort be applied easily to sort items 
within a linked list? 
Is Bubble Sort a “stable” sort? 


void bubbleSort(int Arr[], 


do 
{ 


atLeastOneSwap = false; 


0; 


for (int j j < (n-1); 


{ 
if (Arr[j] > Arr[j + 1]) 

{ 
Swap (Arr[j] ,Arr[jt+1]) ; 
atLeastOneSwap = true; 


} 


} 


int n 
{ 
bool atLeastOneSwap ; 


} 
while (atLeastOneSwap == true) jp 


Is Bubble Sort ever faster than O(n?)? 
When might you use 
Bubble Sort? 


d 


Start by assuming 
that we won't 
do any swaps 


j++) 
Compare each element 
with its neighbor and 
swap them if they're 
out-of-order. 


Don't forget-we swapped! 


If we swapped at least 
once, then start back at 
the top and repeat the 
whole process. 


“Sorting Challenge 


Consider the following array 
of integers: 


By one round, I 
mean one full trip 
through the sort's 2 5 9 
while/for loop. 


147 3 


hich has been sorted by 
one round of either selection 
sort, insertion sort or bubble 
sort. 


Which of these sorts could NOT 
have been used on this array? 
Why? 


selectionSort 
For each of the N books 
Find the smallest book between 
slots iand N 
Swap this smallest book with 
book i 


insertionSort 


s=2 

While books need sorting: 
Focus on the first s books 
If the last book in set is in the 
wrong order THEN 


A. Remove it from shelf 
B. Shift the books to 
the right as required 
C. Insert our book into the 
proper slot 
s=st+l 


bubbleSort 
while the shelf isn't sorted 
repeatedly swap adjacent books 
if they're out of order 


Appendix 


> Shellsort - this won't be on your exam 


The sort 


Shellsort is based on an underlying procedure 
called h-sorting. Let's learn h-sorting first... 


The method for h-sorting an array is simple: 


Pick a value of h 
For each element in the array: 
- If Ali] and A[i+h] are out of order then 
- Swap the two elements 


g 


Errr.... 
“I want h=3" 


» 


d 


If you swapped any elements during the last pass, 
then repeat the entire process again (same h value). 


The Shellsort: h-sorting 


Pick a value of h 
For each element in the array: Lets oe this aie =e 
` If A[i]and Alith] are out of order The she R oe Ing. 
- Swap the two elements ae 
If you swapped any elements, 
repeat the entire process again. 
i i+3 


y Y 


The Shellsort: h-sorting 


Pick a value of h 
For each element in the array: Lets oe this aie =e 
` If Ali]and Alith] are out of order The she R ne Ing. 
- Swap the two elements Seti 
If you swapped any elements, 
repeat the entire process again. 
i i+3 


y Y 
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This time we had no swaps! 
Our array is now 3-sorted! 


is smaller than 
the element 3 items later 
= in the array. 


Of course, it's not completely 
sorted yet, just 3-sorted! 
\ 


This means that every element 


-sorting 


t's 3-sort this array so 
e shells are ascending. 
e.g., h=3 


Question: 

If you 1-sort an array, 
which other sort 
algorithm does this 
remind you of? 


The Shellsort 


The overall Shellsort works as foll 


It's required 

to always end 
with h=1! 

~= eg.5,3,lor ` 
=. 10,741 


Step 1: 
Select a sequence of decreasing h-values, 
ending with an h-value of 1. e.g. 8,4,2,1. 


Step 2: 
First completely 8-sort the array... 
Then completely 4-sort the array... 
Then completely 2-sort the array... 
Finally, completely bubble sort the array... 


and the array's now fully sorted! 


Each h-sort more correctly sorts the array, making 
the process simpler each iteration. 


Shell Sort Questions 


Can Shell Sort be applied easily to 
sort items within a linked list? 


Is Shell Sort a “stable” sort? 
What's the Big-O of Shell Sort? 


When might you use Shell Sort? 


The Shellsort 


Let's do an example on the board: 


Shellsort the following array using h values of: 3, 2, and 1. 


9) |5) |2) 14 |3) |7 


71 . selectionSort 
For each of the N books 
So rfi ng Chal lenge Find the smallest book between 
slots i and N 


Given the following numbers, Swap this smallest book with 


show what they would look La 


like after one, two and three E E 


outer-loop iterations of rd eer 
; i ile books need sorting: 
selection sort, insertion sort NE he eee 
and bubble sort: If the last book in set is in the 
wrong order THEN 
A. Remove it from shelf 
9/52 14 3) 7 B. Shift the books to 


the right as required 
C. Insert our book into the 
proper slot 
s=s+1 


bubbleSort 
while the shelf isn't sorted 
repeatedly swap adjacent books 
if they're out of order 


